Generation of three-dimensional images with digital magnification

ABSTRACT

A system for generating three-dimensional (3D) images from captured images of a target when executing digital magnification. A controller executes a digital magnification on the first image of the target captured by the first image sensor and on the second image captured by the second image sensor of the target. The controller crops the first image and the second image to overlap a first portion of the target captured by the first image sensor with a second portion of the target captured by the second image sensor. The controller adjusts the cropping of the first image and the second image to provide binocular overlap of the first portion of the target with the second portion of target. The displayed cropped first image and the cropped second image display the 3D image at the digital magnification to the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. Nonprovisional Application which claims thebenefit of U.S. Provisional Application No. 63/029,831 filed on May 26,2020, which is incorporated herein by reference in its entirety.

BACKGROUND Field of Disclosure

The present disclosure relates to the generation of theThree-Dimensional (3D) images and specifically to the generation of 3Dimages from the digital magnification of images captured of a target.

Related Art

Conventionally, surgical loupes have been used extensively in varioustypes of surgeries. Surgical loupes are a pair of optical magnifiersthat magnify the surgical field and provide magnified stereoscopicvision. However, conventional surgical loupes have significantlimitations. For example, a single set of conventional surgical loupesonly offer a fixed level of magnification, such as 2× without anycapabilities to vary such magnification. Therefore, surgeons typicallyrequire several pairs of surgical loupes with each pair having adifferent level of magnification to cater for different levels ofmagnifications. Changing surgical loupes in the operating room isinconvenient with an increased cost to have several sets of surgicalloupes with different magnifications customized for a single onesurgeon.

However, equipping conventional surgical loupes with magnifying lensestypically include an increased length resulting in an increased formfactor and increased weight and thereby limit the magnification level.The increased form factor and increased weight also limit the durationof surgical procedures that the surgeon may execute. Further,conventional surgical loupes implement a non-imaging configuration,whereby the magnification lenses magnify and form a pair of virtualimages thereby decreasing the working distances and depths of focus forthe surgeon. Therefore, the surgeon has to restrict the position oftheir head and neck to a specific position as they use the conventionalsurgical loupes. This results in neck pains and cervical diseases forsurgeons with long term use of conventional surgical loupes.

Rather than simply having surgical loupes use non-imagingconfigurations, conventional imaging configurations in the non-surgicalspace include stereo imaging systems and imaging systems with zoomlenses where such conventional imaging configurations generate 3D imageswhile enabling the adjustment of magnification. However, theincorporation of such conventional imaging configurations in thesurgical space require the implementation of two displays and/or zoomlenses for the surgeon. The two stereo displays included in suchconventional stereo imaging systems must be mechanically adjusted foreach magnification level as well as calibrated. Such mechanicaladjustment and calibration in the surgical space is not feasible. Thechanging in magnification for two conventional zoom lenses requires eachimage at each magnification level to always be captured at the center ofthe initial image where each level of magnification continues to capturethe center of the initial image. The resulting 3D image displayed to thesurgeon is significantly skewed thereby preventing the incorporation ofconventional zoom lenses into the surgical space.

BRIEF DESCRIPTION OF THE DRAWINGS/FIGURES

Embodiments of the present disclosure are described with reference tothe accompanying drawings. In the drawings, like reference numeralsindicate identical or functionally similar elements. Additionally, theleft most digit(s) of a reference number typically identifies thedrawing in which the reference number first appears.

FIG. 1A illustrates a schematic view of binocular overlap of human eyesconfiguration where the region seen by both eyes is the overlappedregion included in the scene seen by both eyes;

FIG. 1B illustrates a block diagram of a two imaging sensorconfiguration where two image sensors with two lenses are used in aside-by-side configuration;

FIG. 1C illustrates a block diagram a binocular overlap of two imagingsensor configuration with the regions seen by both imaging sensors isthe overlapped region;

FIG. 2 depicts a schematic view of a conventional digital zoomconfiguration where the original image is cropped and resized (from leftto right);

FIG. 3 illustrates a block diagram of a digital magnification of a 3Dimage system that may generate 3D images when executing digitalmagnification on captured images of a target;

FIG. 4 depicts a schematic view of a conventional digital zoomconfiguration where the zoomed left images and zoomed right images aremisaligned leading to poor 3D vision and depth perception;

FIG. 5 depicts a schematic diagram of a digital magnification withbinocular vertical alignment preservation configuration where themagnified left images and the magnified right images are verticallyaligned thereby resulting in increased 3D visualization;

FIG. 6 depicts a schematic view of a digitally magnified stereo imageswith preservation of vertical alignment configuration whereas thedigital magnification is applied, binocular overlap between the croppedleft images and cropped right images gradually decreases;

FIG. 7 depicts a schematic view of a preservation of binocular overlapand binocular vertical alignment configuration where at 2.3×, 5.3C, and12×, respectively, the left cropped images and the right cropped imageshave binocular overlap of 75% and vertical alignment thereby resultingin an increased 3D visualization experience and depth perception may beprovided to the user; and

FIG. 8 depicts a schematic view a physical embodiment of a digitalmagnification surgical loupe configuration.

DETAILED DESCRIPTION OF THE PRESENT DISCLOSURE

The following Detailed Description refers to accompanying drawings toillustrate exemplary embodiments consistent with the present disclosure.References in the Detailed Description to “one exemplary embodiment,” an“exemplary embodiment,” an “example exemplary embodiment,” etc.,indicate the exemplary embodiment described may include a particularfeature, structure, or characteristic, but every exemplary embodimentmay not necessarily include the particular feature, structure, orcharacteristic. Moreover, such phrases are not necessarily referring tothe same exemplary embodiment. Further, when a particular feature,structure, or characteristic may be described in connection with anexemplary embodiment, it is within the knowledge of those skilled in theart(s) to effect such feature, structure, or characteristic inconnection with other exemplary embodiments whether or not explicitlydescribed.

The exemplary embodiments described herein are provided for illustrativepurposes, and are not limiting. Other exemplary embodiments arepossible, and modifications may be made to the exemplary embodimentswithin the spirit and scope of the present disclosure. Therefore, theDetailed Description is not meant to limit the present disclosure.Rather, the scope of the present disclosure is defined only inaccordance with the following claims and their equivalents.

Embodiments of the present disclosure may be implemented in hardware,firmware, software, or any combination thereof. Embodiments of thepresent disclosure may also be implemented as instructions applied by amachine-readable medium, which may be read and executed by one or moreprocessors. A machine-readable medium may include any mechanism forstoring or transmitting information in a form readable by a machine(e.g., a computing device). For example, a machine-readable medium mayinclude read only memory (“ROM”), random access memory (“RAM”), magneticdisk storage media, optical storage media, flash memory devices,electrical optical, acoustical or other forms of propagated signals(e.g., carrier waves, infrared signals, digital signals, etc.), andothers. Further firmware, software routines, and instructions may bedescribed herein as performing certain actions. However, it should beappreciated that such descriptions are merely for convenience and thatsuch actions in fact result from computing devices, processors,controllers, or other devices executing the firmware, software,routines, instructions, etc.

For purposes of this discussion, each of the various componentsdiscussed may be considered a module, and the term “module” shall beunderstood to include at least one software, firmware, and hardware(such as one or more circuit, microchip, or device, or any combinationthereof), and any combination thereof. In addition, it will beunderstood that each module may include one, or more than one, componentwithin an actual device, and each component that forms a part of thedescribed module may function either cooperatively or independently fromany other component forming a part of the module. Conversely, multiplemodules described herein may represent a single component within anactual device. Further, components within a module may be in a singledevice or distributed among multiple devices in a wired or wirelessmanner.

The following Detailed Description of the exemplary embodiments will sofully reveal the general nature of the present disclosure that otherscan, by applying knowledge of those skilled in the relevant art(s),readily modify and/or adapt for various applications such exemplaryembodiments, without undue experimentation, without departing from thespirit and scope of the present disclosure. Therefore, such adaptationsand modifications are intended to be within the meaning and plurality ofequivalents of the exemplary embodiments based upon the teaching andguidance presented herein. It is to be understood that the phraseologyor terminology herein for the purpose of description and not oflimitation, such that the terminology or phraseology of the presentspecification is to be interpreted by those skilled in the relevantart(s) in light of the teachings herein.

System Overview

FIG. 1A illustrates a schematic view of binocular overlap of human eyesconfiguration 100 where the region seen by both eyes is the overlappedregion included in the scene seen by both eyes. The binocular overlap ofhuman eyes configuration 100 includes a right eye 110 a, a left eye 110b, an image as seen by right eye 120 a, an image as seen by left eye 120b, and a binocular overlap 120 c as seen by both eyes.

The present invention describes the apparatus, systems, and methods forconstructing augmented reality devices for medical and dentalmagnification. One of the key concepts in 3D imaging and visualizationis binocular overlap 120 c. Binocular overlap 120 c describes theoverlap between the image as seen by the left eye 120 b, versus theimage as seen by the right eye 120 a. For human being, a binocularoverlap 120 c is approximately 70%.

FIG. 1B illustrates a block diagram of a two imaging sensorconfiguration 150 where two image sensors with two lenses are used in aside-by-side configuration. The two imaging sensor configuration 150includes a right image sensor 130 a, a left image sensor 130 b, a rightlens 140 a, and a left lens 140 b. FIG. 1C illustrates a block diagram abinocular overlap of two imaging sensor configuration 175 with theregions seen by both imaging sensors is the overlapped region. Thebinocular overlap of two imaging sensor configuration 175 includes acaptured region by right image sensor 150 a, a captured region by leftimage sensor 150 b, and a binocular overlap region 150 c. FIG. 1Cdepicts the binocular overlap region 150 c that is generated when aright image sensor 130 a and a left image sensor 130 b are used in aside-by-side configuration as depicted in FIG. 1B.

FIG. 2 depicts a schematic view of a conventional digital zoomconfiguration 200 where the original image is cropped and resized (fromleft to right). The cropped and resized images are displayed to the userafter conventional digital zooming. Conventionally, digital zoom hasbeen commonly used to zoom the image. The principle of conventionaldigital zoom is illustrated in FIG. 2. Although conventional digitalzoom can magnify the images without the need of zoom lenses, it is notsuitable for 3D magnification.

FIG. 3 illustrates a block diagram of a digital magnification of a 3Dimage system 300 that may generate 3D images when executing digitalmagnification on captured images of a target. The digital magnificationof a 3D image system 300 includes a right lens 340 a, a left lens 340 b,a right image sensor 330 a, a left image sensor 330 b, a controller 310,a near-eye 3D display 320, and an eyeglass frame 350. In one embodiment,the eyeglass frame 350 is a head mount. In another embodiment, theeyeglass frame 350 is a traditional eyeglass frame sitting on the noseand ears of a user.

The digital magnification of a 3D image system 300 may generate 3Dimages from captured images of a target when executing digitalmagnification on the captured images to maintain the 3D images generatedof the target after digital magnification. A first image sensor (such asright image sensor 330 a) may capture a first image at an original sizeof the target. A second image sensor (such as left image sensor 330 b)may be positioned on a common x-axis with the first image sensor 330 ato capture a second image at the original size of the target. It shouldbe appreciated that the first image sensor 330 a and the second imagesensor 330 b may be positioned with either a converging angle or adiverging angle.

A controller 310 may execute a digital magnification on the first imagecaptured by the first image sensor 330 a at the original size of thetarget and on the second image captured by the second image sensor 330 bat the original size of the target. The controller 310 may crop thefirst image captured by the first image sensor 330 a and the secondimage captured by the second image sensor 330 b to overlap a firstportion of the target captured by the first image sensor 330 a with asecond portion of the target captured by the second image sensor 330 b.The first portion of the target captured by the first image sensor 330 aoverlaps with the second portion of the target captured by the secondimage sensor 330 b. In one aspect, the first image sensor 330 a isfurther coupled with a first autofocus lens and the second image sensor330 b is further coupled with a second autofocus lens. The autofocuslenses may enable autofocus.

The controller 310 may adjust the cropping of the first image and thesecond image to provide binocular overlap of the first portion of thetarget with the second portion of the target. The binocular overlap ofthe first image and the second image is an overlap threshold that whensatisfied results in a 3D image of the target displayed to a user afterthe digital magnification is executed. The controller may instruct adisplay (such as near-eye 3D display 320) to display the cropped firstimage and the cropped second image that includes the binocular overlapto the user. The displayed cropped first image and the cropped secondimage display the 3D image at the digital magnification to the user.

The controller 310 may resize the cropped first image to the originalsize of the first image captured by the first image sensor 330 a and thecropped second image to the original size of the second image capturedby the second image sensor 330 b. The cropped first image as resized andthe cropped second image resized includes the binocular overlap of thefirst image and the second image. The controller 310 may instruct thenear-eye 3D display 320 to display the resized and cropped first imageand the resized and cropped second image that includes the binocularoverlap to the user. The displayed resized and cropped first image andthe resized and cropped second image display the 3D image at the digitalmagnification to the user. It should be appreciated that in oneembodiment the controller 310 may crop the first image captured by thefirst image sensor 330 a, to generate both left cropped image and rightcropped image. In this embodiment, the second image captured by thesecond image sensor 330 b is not used.

In one aspect, the display 320 is a near-eye display. In one embodiment,the display 320 is a 2D display. In another embodiment, the display 320is a 3D display. It should be further appreciated that the near-eyedisplay 320 may comprise LCD (liquid crystal) microdisplays, LED (lightemitting diode) microdisplays, organic LED (OLED) microdisplays, liquidcrystal on silicon (LCOS) microdisplays, retinal scanning displays,virtual retinal displays, optical see-through displays, videosee-through displays, convertible video-optical see-through displays,wearable projection displays, projection display, and the like. Itshould be the appreciated that the display 320 may be stereoscopic toenable displaying of 3D content. In another embodiment, the display 320is a projection display. It should be appreciated that the display 320may be a monitor placed near the user.

It should be further appreciated that the display 320 may be a 3Dmonitor placed near the user and the user will wear a polarizing glassor active shutter glasses. It should be further appreciated that thedisplay 320 may be a half transparent mirror placed near the user toreflect the image projected by a projector. It should be further beappreciated that the said projector may be 2D or 3D. It should befurther appreciated that the said projector may be used with the userwearing a polarizing glass or active shutter glasses. In one embodiment,the display 320 is a flat panel 2D monitor or TV. In another embodiment,the display 320 is a flat panel 3D monitor or 3D TV. The 3D monitor/TVmay need to work with passive polarizers or active shutter glasses. Inone aspect, the 3D monitor/TV is glass-free. It should be appreciatedthat the display 320 can be a touchscreen, or a projector. In oneexample, the display 320 comprises a half transparent mirror that canreflect projection of images to the eyes of the user. The images beingprojected may be 3D, and the user may wear 3D glasses (e.g. polarizer;active shutter 3D glasses) to visualize the 3D image data reflected bythe half transparent mirror. The half transparent mirror may be placedon top of the surgical field to allow the user to see through the halftransparent mirror to visualize the surgical field.

It should be appreciated that the binocular of the system may be set ashigh as 100% or as low as 0%, depending on the specific application. Inone aspect, the binocular overlap is set to be within the range of 60%and 100%. In another aspect, the binocular overlap is dynamic and notstatic.

In one aspect, the digital magnification of a 3D image system 300 mayfurther comprise additional sensors or components. In one embodiment,the system 300 further comprise a microphone, which may enable audiorecording and/or communication. In one embodiment, the system 300further comprise a proximity sensor, which may sense if user is wearingthe system. In another embodiment, the system 300 further comprise ainertial measurement unit (IMU), an accelerometers, a gyroscopes, amagnetometers, or a combination thereof. In one embodiment, the system300 further comprise a loudspeaker or earphone, which may enable audioreplay or communication.

It should be further appreciated that the system can be applied avariety of applications, including but not limited to surgical, medical,veterinary, military, tactical, educational, industrial, consumer,jewelry fields.

Digital Magnification With Binocular Vertical Alignment

FIG. 4 depicts a schematic view of a conventional digital zoomconfiguration 400 where the zoomed left images and zoomed right imagesare misaligned leading to poor 3D vision and depth perception. Theconventional digital zoom configuration 400 includes the zoomed rightimages 410 a that are misaligned with the zoomed left images 410 b.Conventional digital zoom does not work well on magnifying ofstereo-images for 3D display. FIG. 3 shows an example of directapplication of conventional digital zoom to stereo-images. Conventionaldigital zoom is not suitable for magnifying 3D stereo-images, as itintroduces binocular vertical misalignment.

The controller 310 may crop the first image captured by the first imagesensor 330 a and the second image captured by the second image sensor330 b to vertically align the overlap of the first portion of the targetwith the second portion of the target. The cropped first image is invertical alignment of the cropped second image when each verticalcoordinate of the cropped first image is aligned with each correspondingvertical coordinate of the cropped second image. The controller 310 mayadjust the cropping of the first image and the second image to providebinocular overlap of the first portion of the target with the secondportion of the target. The binocular overlap of the first image and thesecond image is vertically aligned to satisfy the overlap threshold togenerate the 3D image of the target displayed to the user after thedigital magnification is executed.

The present invention discloses a digital magnification method that alsoensures binocular vertical alignment. In one embodiment, the left imageis captured by the left image sensor 330 b and cropped by the controller310, and the right image is captured by the right image sensor 330 a andcropped by the controller 310, while the cropping of left and rightimages preserves vertical alignment. The left and right images arecropped in such a way the vertical coordinates of the cropped left imageand the vertical coordinates cropped right image are aligned.

In an embodiment, the left image sensor 330 b with the left lens 340 bthat are worn by the user may capture a left image. The right imagesensor 330 a with the right lens 340 a that are worn by the user maycapture a right image. The left image and the right image may beprovided to the controller 310. The controller 310 may crop the leftimage to generate a cropped left image. The controller 310 may crop theright image to generate a cropped right image and may preserve thevertical alignment of the cropped right image with respect to thecropped left image. The controller 310 may resize the cropped left imageto generate a cropped and resized left image. The controller 310 mayresize the cropped right image to generate a cropped and resized rightimage. The near-eye 3D display 320 worn by the user may display thecropped and resized left image to the left eye of the user. The near-eye3D display 320 worn by the user may display the cropped and resizedright image to the right eye of the user. It should be appreciated thatthe controller can be a microcontroller, a computer, afield-programmable gate array (FPGA), an application specific integratedcircuits (ASIC), or a combination thereof.

In one embodiment, the left image sensor and right image sensor areidentical image sensors. The image sensors may use the same type ofimage lenses. The left and right image sensors may be placed andcalibrated, so that the left image captured and right image captured arevertically aligned, prior to any digital magnification process. Thedigital magnification process preserve the vertical alignment. Forexample, assuming the left image and right image each have 800(horizontal, column) by 600 (vertical, row) pixels. After digitalmagnification, the row 201 to row 400 of pixels of left image togenerate a cropped left image, and the row 201 to row 400 of pixels ofthe right image are used to generate a cropped right image. Therefore,the vertical alignment is preserved.

In one embodiment, the left image sensor and right image sensor are notidentical image sensors. In this case, the left image captured and rightimage captured are first calibrated and aligned vertically, prior to anydigital magnification process. For example, assuming the left imagecaptured by the left image sensor have 800 (horizontal, column) by 600(vertical, row) pixels, but the right image captured by the right imagesensor have 400 (horizontal) by 300 (vertical) pixels. The left imageand right image are first vertically aligned. For instance, the row #0,200, 400, 600 of the left image may correspond to the row #0, 100, 200,300 of the right image, respectively. After digital magnification, asubset of the row 200 to row 400 of pixels of left image, and a subsetof the row 100 to row 200 of pixels of the right image are used.Therefore, the vertical alignment is preserved.

FIG. 5 depicts a schematic diagram of a digital magnification withbinocular vertical alignment preservation configuration 500 where themagnified left images and the magnified right images are verticallyaligned thereby resulting in increased 3D visualization. The digitalmagnification with binocular vertical alignment preservationconfiguration 500 includes the zoomed right images 510 b are verticallyaligned with the zoomed left images 510 a thereby resulting in increased3D visualization.

Digital Magnification With Preservation of Binocular Overlap

FIG. 6 depicts a schematic view of a digitally magnified stereo imageswith preservation of vertical alignment configuration 600 whereas thedigital magnification is applied, binocular overlap between the croppedleft images and cropped right images gradually decreases. The digitallymagnified stereo images with preservation of vertical alignmentconfiguration 600 includes digitally magnified right images 610 a arevertically aligned with the digitally magnified left images 610 b. Forexample, at a 2.3× magnification, the binocular overlap decreases from75% to 50% resulting in a decrease in 3D visualization. At a 5.3×magnification, the binocular overlap decreases from 75% to 0%. Thevertical alignment preservation without the preservation of binocularoverlap may result in the gradual decrease in binocular overlap witheach digital magnification.

After executing a first digital magnification at a first digitalmagnification level on the first image captured by the first imagesensor 330 b and on the second image captured by the second image sensor330 a, the controller 310 may maintain the binocular overlap generatedby adjusting the cropping of the first image and the second image tosatisfy the overlap threshold. In one aspect, during the digitalmagnification process a fixed binocular overlap number is maintained,such as 80%, 90% or 100%. In another aspect, during the digitalmagnification process a range of binocular overlap number is maintained,such as 60%-90%.

The controller 310 may execute a second digital magnification at asecond digital magnification level on the first image captured by thefirst image sensor 330 a and the second image captured by the secondimage sensor 330 b. The second digital magnification level is increasedfrom the first digital magnification level. The controller 310 maymaintain the binocular overlap generated after executing the firstdigital magnification at the first digital magnification level on thefirst image and the second image when executing the second digitalmagnification at the second digital magnification level.

After executing each previous digital magnification at each previousdigital magnification level on the first image and the second image, thecontroller 310 may maintain the binocular overlap and the verticalalignment determined when executing the first digital magnification atthe first digital magnification level on the first image and the secondimage. The controller 310 may continue to maintain the binocular overlapand the vertical alignment determined from the adjusting of the croppingof the first image and the second image to satisfy the overlap thresholdafter executing the first digital magnification at the first digitalmagnification level on the first image and the second image for eachsubsequent digital magnification level. Each subsequent digitalmagnification level is increased from each previous digitalmagnification level. For example, the overlap threshold may be satisfiedwhen the binocular overlap includes 75% overlap of the first image andthe second image is maintained for each subsequent digital magnificationat each subsequent digital magnification level. In one embodiment, eachsubsequent digital magnification from the previous magnification level(e.g. increase from 1× to 2×, and increase 2× to 4×) may be a recursivefunction.

The controller 310 may execute first digital magnification at the firstdigital magnification level on a non-concentric portion of the firstimage and a non-concentric portion of the second image. Thenon-concentric portion of the first image and the second image is aportion of the first image and the second image that differs from acenter of the first image and the second image. The controller 310 mayadjust the cropping of the first image and the second image to providebinocular overlap of the non-concentric portion of the first image andthe non-concentric portion of the second image. The binocular overlap ofthe non-concentric portion of the first image and the non-concentricportion of the second image satisfies the overlap threshold eitherspecified as a fixed number or a range. The controller 310 may continueto crop a non-concentric portion of the first image and a non-concentricportion of the second image for each subsequent digital magnification ateach subsequent digital magnification level. The binocular overlap ofthe non-concentric portion of the first image and the non-concentricportion of the second image is maintained from the first digitalmagnification at the first digital magnification level.

The non-concentric portion of the first image and the non-concentricportion of the second image may be resized to display to the user. Inone aspect, at each magnification level, a first center of cropping ofthe non-concentric portion of the first image and a second center ofcropping of the non-concentric portion of the second image aredetermined by the system 300. In one embodiment, the first center ofcropping is fixed at the particular part of the first image, and secondcenter of cropping at each magnification level is determined based onthe location of the corresponding first center of cropping and thetargeted binocular overlap. It should be appreciated that in someembodiment and at one or more magnification level, the digitalmagnification on either left image or right image may be concentric. Forexample, digital magnification on the left image is concentric but thedigital magnification on the right image is non-concentric to maintainthe binocular overlap.

In one embodiment, the left image sensor and right image sensor areidentical image sensors. The image sensors may use the same type ofimage lenses, including autofocus lenses. The left and right imagesensors may be placed and calibrated, so that the left image capturedand right image captured are vertically aligned, prior to any digitalmagnification process. The digital magnification process preserves thevertical alignment and binocular overlap (e.g. 80%). For example,assuming the left image and right image each have 800 (horizontal,column) by 600 (vertical, row) pixels. After digital magnification, thepixels from row 201 to row 400 and column 401 to column 600 of leftimage are used to generate a cropped left image, and the row 201 to row400 and column 201 to column 400 of right image are used to generate acropped right image. This cropping may generate a satisfactory binocularoverlap (e.g. 80%). The non-concentric cropping in the digitalmagnification combined with resizing may enable magnification whilepreserving of both binocular overlap and vertical alignment. Similarly,when the system increase to a higher digital magnification level,further non-concentric cropping on at least one of the image (e.g. leftor right images) are performed in conjunction with resizing to enablemagnification while preserving of both binocular overlap and verticalalignment

In another example, machine learning algorithms are used for determininga center of cropping for the left image, or a center of cropping for theright image, or both centers, during the digital magnification process.In one aspect, object recognition and localization based on machinelearning (e.g. recognize surgical field, or recognize surgicalinstrument, or recognize tissues, etc.) may determine at least onecenter of the cropping. For example, the surgical bed is recognized andlocalized based on the left image, and a location within the surgicalbed (e.g. centroid) is assigned to be the center of cropping for theleft image, and the center of cropping for the right image is calculatedbased on the center of cropping for the left image and the desirablebinocular overlap to be maintained.

In one aspect, supervised learning can be implemented. In anotheraspect, unsupervised learning can be implemented. In yet another aspect,reinforcement learning can be implemented. It should be appreciated thatfeature learning, sparse dictionary learning, anomaly detection,association rules may also be implemented. Various models may beimplemented for machine learning. In one aspect, artificial neuralnetworks are used. In another aspect, decision trees are used. In yetanother aspect, support vector machines are used. In yet another aspect,Bayesian networks are used. In yet another aspect, genetic algorithmsare used.

In yet another example, neural networks, convolutional neural networks,or deep learning are used for object recognition, image classification,object localization, image segmentation, image registration, or acombination thereof. Neural network based systems are advantageous inmany cases for image segmentation, recognition and registration tasks.

In one example, U-Net is used, which has a contraction path andexpansion path. The contraction path has consecutive convolutionallayers and max-pooling layer. The expansion path performs up-conversionand may have convolutional layers. The convolutional layer(s) prior tothe output maps the feature vector to the required number of targetclasses in the final segmentation output. In one example, V-net isimplemented for image segmentation to isolate the organ or tissue ofinterest (e.g. vertebral bodies). In one example, Autoencoder based DeepLearning Architecture is used for image segmentation to isolate theorgan or tissue of interest. In one example, backpropagation is used fortraining the neural networks.

In yet another example, deep residual learning is performed for imagerecognition or image segmentation, or image registration. A residuallearning framework is utilized to ease the training of networks. Aplurality of layers is implemented as learning residual functions withreference to the layer inputs, instead of learning unreferencedfunctions. One example of network that performs deep residual learningis deep Residual Network or ResNet.

In another embodiment, a Generative Adversarial Network (GAN) is usedfor for image recognition or image segmentation, or image registration.In one example, the GAN performs image segmentation to isolate the organor tissue of interest. In the GAN, a generator is implemented throughneural network to models a transform function which takes in a randomvariable as input and follows the targeted distribution when trained. Adiscriminator is implemented through another neural networksimultaneously to distinguish between generated data and true data. Inone example, the first network tries to maximize the finalclassification error between generated data and true data while thesecond network attempts to minimize the same error. Both networks mayimprove after iterations of the training process.

In yet another example, ensemble methods are used, wherein multiplelearning algorithms are used to obtain better predictive performance. Inone aspect, Bayes optimal classifier is used. In another aspect,bootstrap aggregating is used. In yet another aspect, boosting is used.In yet another aspect, Bayesian parameter averaging is used. In yetanother example, Bayesian model combination is used. In yet anotherexample, bucket of models is used. In yet another example, stacking isused. In yet another aspect, a random forests algorithm is used. In yetanother aspect, a gradient boosting algorithm is used.

The controller 310 may determine a distance that the first image sensor330 b and the second image sensor 330 a is positioned from the target.The controller 310 may execute the cropping of the first image and thesecond image to maintain the vertical alignment and the binocularoverlap for each digital magnification at each digital magnificationlevel based on the distance of the first image sensor 330 b and thesecond image sensor 330 a is from the target.

In another embodiment, the system allows the user to determine a centerof cropping for the left image, or a center of cropping for the rightimage, or both centers, for the digital magnification process. In thecase of many users, each may have their own settings.

The display 320 may include one of a plurality of wearable display thatdisplays the resized and cropped first image and the resized and croppedsecond image to display the 3D image of the target after the digitalmagnification is executed that includes the binocular overlap of thefirst image and the second image that are vertically aligned to satisfythe overlap threshold. In one aspect, the first image sensor 330 b andthe second image sensor 330 a may be positioned proximate the display320 for the user to execute a surgical procedure on the target that is apatient. In another aspect, the first image sensor 330 b and the secondimage sensor 330 a may be positioned close to the display 320 for theuser to execute a surgical procedure on the target that is a patient. Inanother example, the first image sensor 330 b and the second imagesensor 330 a may be positioned on a stand, not adjacent to the display320. It should be appreciated that the said stand may be motorized orhas a robot. The display 320 may be a 3D monitor, a 3D projector, or a3D projector with a combiner, used with 3D glasses (e.g. polarizers oractive shutter glasses).

The present invention discloses a method for digitally magnifying theimages, while preserving the binocular overlap. In one embodiment, thecropping of left image and cropping of right image may be performed bythe controller 310 with the binocular overlapped preserved. For example,if the original left image and right image have an original binocularoverlap of 75%, the cropped left image and cropped right image may becropped by the controller 310 in such a way so that the binocularoverlap of cropped images will also be 75%.

In an embodiment, the left image sensor 330 b with the left lens 340 bthat are worn by the user may capture a left image. The right imagesensor 330 a with the right lens 340 a that are worn by the user maycapture a right image. The left image and the right image may beprovided to the controller 310. The controller 310 may calculate a leftcrop function that specifies how to crop the left image and a right cropfunction that specifies how to crop the right image. The left cropfunction and the right crop function preserve binocular overlap andbinocular vertical alignment. The controller 310 may crop the left imageto generate a cropped left image using the left crop function thatpreserves binocular overlap and binocular vertical alignment. Thecontroller 310 may crop the right image to generate a cropped rightimage using the right crop function that preserves binocular overlap andbinocular vertical alignment.

The controller 310 may resize the cropped left image to generate acropped and resized left image. The controller 310 may resize thecropped right image to generate a cropped and resized right image. Thedisplay 320 worn by the user may display the cropped and resized leftimage to the left eye of the user. The display 320 may display thecropped and resized right image to the right eye of the user. In oneaspect, the display 320 may be a near-eye 3D display. In another aspect,the display 320 may be a 3D monitor, a 3D projector, or a 3D projectorwith a combiner, used with 3D glasses (e.g. polarizers or active shutterglasses).

FIG. 7 depicts a schematic view of a preservation of binocular overlapand binocular vertical alignment configuration 700 where at 2.3×, 5.3×,and 12× magnifications, respectively, the left cropped images and theright cropped images have binocular overlap of 75% and verticalalignment thereby resulting in an increased 3D visualization experienceand depth perception may be provided to the user. The preservation ofbinocular overlap and binocular vertical alignment configuration 700includes right cropped images 710 b and left cropped images 710 a.

In another embodiment, the digital magnification method furthercomprises of an additional condition to satisfy: the left cropped imageshares the same geometrical center as that of the left original image.The right cropped image may be calculated by the controller 310 andgenerated accordingly by the controller 310 based on the cropping of theleft cropped image, while preserving the binocular overlap and binocularvertical alignment. The benefit of this implementation is: the digitalmagnification process may be coaxial along the center of the left image(the optical axis), and the progression of digital magnification mayalign with the line of sight of the user's left eye. Alternatively, thecropped right image may share the same center as the right originalimage. The left cropped image may be calculated by the controller 310and generated accordingly by the controller 310 based on the positionand cropping of the right cropped image, while preserving the binocularoverlap and binocular vertical alignment.

In another embodiment, the acceptable binocular overlap of croppedimages may be specified as a range, rather than a specific number. Forinstance, the binocular overlap of cropped left and right images may bespecified to be within a range between 60% to 90%. Any number between60% and 90% may be considered satisfactory for digital magnification.With an acceptable range of binocular overlap as a guideline forcropping left and right images, the left image sensor 330 b with theleft lens 340 b that are worn by the user controller 310 may capture aleft image. The right image sensor 330 a and the right lens 340 a thatare worn by the user may capture a right image. The left image and theright image may be provided to the controller 310.

The controller 310 may calculate a left crop function that specifies howto crop the left image and the right crop function that specifies how tocrop the right image. The left crop function and the right crop functionmay preserve binocular vertical alignment. The left crop function andthe right crop function may preserve binocular overlap as specified by arange of acceptable binocular overlap, such as 60% to 90%. Thecontroller 310 may crop the left image to generate a cropped left imageusing the left crop function that preserves binocular overlap andbinocular vertical alignment. The controller 310 may crop the rightimage to generate a cropped right image using the right crop functionthat preserves binocular overlap and binocular vertical alignment. Thecontroller 310 may resize the cropped left image to generate a croppedand resized left image. The controller 310 resizes the cropped rightimage to generate a cropped and resized right image. The display 320 maydisplay the cropped and resized left image to the left eye of the user.The display 320 may display the cropped and resized right image to theright eye of the user.

In another embodiment, the left lens 340 b and right lens 340 a may bezoom lenses. The focal length and angle of view of zoom lenses may bevaried, enabling optical zoom. Therefore, optical zoom may be used inconjunction with the aforementioned digital magnification methods. Forexample, 5.3× digital magnification may be used in conjunction with 2×optical zoom (10.6× magnification in total). It should be appreciatedthat the levels of digital magnification may be either continuous (e.g.magnifying with fine level of increments over a range: e.g. anymagnification level within 2×-7×), or the magnification levels may bediscrete (2×, 2.5×, 3×, 4×, 6×, 7×, etc). In another embodiment, thecontroller 310 may transmit the magnified left image and/or right imageto another 3D display device for visualization. The 3D display devicemay be a wearable display, a monitor, a projector, a projector with acombiner, a passive 3D monitor with 3D polarized glasses, a active 3Dmonitor with active shutter 3D glasses, or a combination thereof. In yetanother embodiment, the controller 310 may transmit the magnified leftimage and/or right image to another computer for visualization, storage,and broadcast. In yet another embodiment, the controller 310 may recordthe magnified left image and/or magnified right image.

In yet another embodiment, the controller 310 may apply computer visionand/or image processing techniques the magnified left image and/ormagnified right image. Additional computer vison analysis can enabledecision support, object recognition, image registration, and objecttracking. For example, deep learning and neural networks may be used. Inyet another embodiment, the near-eye 3D display 320 may display othermedical image data to the user (e.g. CT, MRI, ultrasound, nuclearmedicine, surgical navigation, fluoroscopy, etc) and the other medicalimage data is overlaid with the magnified left image and/or magnifiedright image. It should be further appreciated that more than two imagesensors may be used in the system. In one example, when there are morethan two image sensors, at any given moment, only two image sensors areselected to participate in the digital magnification process. (e.g.three color sensors with three lenses). It should be appreciated that incase of multiple image sensors and image lenses, multiple setsconsisting of two of those sensors may be calibrated with respect toeach other in separate processes.

In one embodiment, only one image sensor is used. This image sensor willserve as both left image sensor 330 a and right image sensor 330 b. Inanother embodiment, a 3D scanning unit comprising of a projector and animage sensor is used, similar to a 3D scanner. A 3D scan can be thusgenerated. The 3D scanning unit may use epipolar geometry for the 3Dscan. By using different virtual viewpoints and projection angles, avirtual left image and virtual right image can be generated based on the3D scan. The digital magnification process aforementioned may be appliedto the virtual left image and virtual right image.

Apparatuses and Systems for Digital Magnification and 3D AugmentedReality Display

FIG. 8 depicts a schematic view a physical embodiment of a digitalmagnification wearable device configuration 800. The digitalmagnification wearable device configuration 800 includes the right imagesensor 330 a, the left image sensor 330 b, the right lens 340 a, theleft lens 340 b, the right near-eye display 320 a, the left near-eyedisplay 320 b, and an eyeglass frame 350. It should be appreciated thatthe wearable frame may be in the form of a head mount, in lieu of aneyeglass frame. It should be appreciated that the controller 310 may bea microcontroller, a computer, an FPGA, or an ASIC. The digitalmagnification wearable device configuration 800 may execute digitalmagnification with preservation of binocular overlap and binocularvertical alignment.

In one embodiment, the digital magnification wearable deviceconfiguration 800 may further include transparent plastic or glass,surrounding the left near eye display 320 b and right near eye display320 a. For example, the digital magnification wearable deviceconfiguration 800 may use a compact offset configuration, whereby only apart of area before each eye is none-transparent and the other parts aretransparent. In one example, the center part of area before each eye isnone-transparent and the peripheral parts are transparent. This way, theuser such as surgeon/dentist can see around the near eye digital displayto look at the patient with unhindered natural vision. In oneembodiment, the digital magnification wearable device configuration 800may further include prescription eyeglasses, so that nearsightedness,farsightedness, and astigmatism may be corrected.

In another embodiment, the digital magnification wearable deviceconfiguration 800 may include an optical see-through configuration. Thenear-eye 3D displays 320(a-b) are both transparent or semi-transparent.In one embodiment, the image sensors 330(a-b) may be a pair of colorimage sensors. Thus, the digital magnification wearable deviceconfiguration 800 may digitally magnify stereoscopic color images anddisplay to the user in the near-eye 3D display 320(a-b) in 3D. In oneexample, the left and right lenses 340(a-b) are lenses with fixed focallengths. In another example, the left and right lenses 340(a-b) are zoomlenses with variable focal lengths. In another example, the color imagesensors may be complementary metal-oxide-semiconductor (CMOS) imagesensors. In yet another example, the color image sensors may becharge-coupled device (CCD) image sensors. In one example, the left andright color image sensors are coupled with autofocus lenes to enableautofocus.

In one embodiment, only one image sensor is used. This image sensor willserve as both left image sensor 330 a and right image sensor 330 b. Inanother embodiment, a 3D scanning unit comprising of a projector and animage sensor is used, similar to a 3D scanner. A 3D scan can be thusgenerated. The 3D scanning unit may use epipolar geometry for the 3Dscan. By using different virtual viewpoints and projection angles, avirtual left image and virtual right image can be generated based on the3D scan. The digital magnification process aforementioned may be appliedto the virtual left image and virtual right image. The 3D scanning unitmay use visible wavelengths, infrared wavelengths, ultravioletwavelengths, or a combination thereof.

The aforementioned 3D scanning unit may project dynamic projectionpattern to facilitate 3D scanning. A few examples of dynamic patternsare binary code, stripe boundary code, and miere pattern. In oneembodiment, binary codeword is represented by a series of black andwhite stripes. If black represents 1 and white represents 0, the seriesof 0 and 1 at any given location may be encoded by the dynamicprojection pattern; the binary dynamic projection pattern may becaptured by the image sensor and lens, and decoded to recover the binarycodeword that encodes an location (e.g. 10100011). In theory, N binarypatterns may generate 2N different codewords per image dimension (x or ydimension). Similarly, binary coding may be extended to N-bits coding.For example, instead of binary case where only 1 and 0 are representedby black and white, a N-bits integer may be represented by an intensityin between. For instance, if it is a 2-bit encoding system, 2*2=4different possibilities. If maximum intensity is I, 0, 1, 2, 3 can berepresented by I, 2/3*I, 1/3*I, and 0, respectively. In other examples,dynamic stripe boundary code-based projection or the dynamic Moirecode-based projection can be implemented.

In another embodiment, dynamic Fourier transform profilometry may beimplemented by 3D scanning unit. In one aspect, periodical signals aregenerated to carry the frequency domain information including spatialfrequency and phase. Inverse Fourier transform of only the fundamentalfrequency results in a principle phase value ranging from −π to π. Afterspatial or temporal phase unwrapping (The process to remove 2πdiscontinuities and generate continuous map), actual 3D shape of patientanatomy may be recovered. Fourier transform profilometry is lesssensitive to the effect of out-of-focus images of patients, making it asuitable technology for intraoperative 3D scanning. Similarly, π-shiftedmodified Fourier transform profilometry may be implementedintraoperatively, where a π-shifted pattern is added to enable the 3Dscanning.

In another example, a DC image may be used with Fourier transformprofilometry in the 3D scanning unit. By capturing the DC component, theDC-modified Fourier transform profilometry may improve 3D scan qualityintraoperatively. In another example, N-step phase-shifting Fouriertransform profilometry may be implemented intraoperatively. It should beappreciated that the larger the number of steps (N) is chosen, thehigher the 3D scanning accuracy. For instance, three-step phase-shiftingFourier transform profilometry may be implemented to enable high speed3D scanning intraoperatively. It should be appreciated that periodicalpatterns such as trapezoidal, sinusoidal, or triangular pattern may beused in the Fourier transform profilometry for intraoperative 3D scan.It should be further appreciated that windowed Fourier transformprofilometry, two-dimensional Fourier transform profilometry, or waveletFourier transform profilometry may also be implemented by theaforementioned apparatuses and systems. It should be appreciated morethan one frequency of periodical signal (e.g. dual frequencies) may beused in the modified Fourier transform profilometry, so that phaseunwrapping become optional in the intraoperative 3D scan. The dynamicFourier transform profilometry and modified Fourier transformprofilometry discussed herein may improve the quality of 3D scan of thepatient. Improved 3D scan may enhance the image registration betweenintraoperative 3D scan and preoperative images (e.g. MRI and CT),thereby improving the surgical navigation.

In yet another embodiment, the aforementioned 3D scanning unitimplements Fourier transform profilometry or modified Fourier transformprofilometry, in combination with binary codeword projection. TheFourier transform profilometry and binary codeword projection may beimplemented sequentially, concurrently, or a combination thereof. Thecombined approach may improve the 3D scanning accuracy, albert at thecost of 3D scanning speed.

In another embodiment, the aforementioned projector may include at leastone lens. The lens is configured such a way so that the projectedpattern(s) are defocused. The defocusing process by the lens is similara convolution of gaussian filter on the binary pattern. Consequently,the defocused binary pattern may create periodical patterns that aresimilar to sinusoidal patterns.

In another example, dithering techniques are used to generatedhigh-quality periodical fringe patterns through binarizing a higherorder bits fringe pattern (e.g. 8 bits) such as sinusoidal fringepatterns. In one example, ordered dithering is implemented; for example,Bayer matrix can be used to enable ordered dithering. In anotherexample, error-diffusion dithering is implemented; for instance,Floyd-Steinberg (FS) dithering or minimized average error dithering maybe implemented. It should be appreciated that in some cases thedithering techniques may be implemented in combination with defocusingtechnique to improve the quality of intraoperative 3D scan.

In another example, the aforementioned projector may generatestatistical pattern. For instance, the projector may generate a pseudorandom pattern that includes a plurality of dots. Each position of eachcorresponding dot included in the pseudo random pattern may bepre-determined by the projector. The projector may project the pseudorandom pattern onto the patient or target. Each position of eachcorresponding dot included in the pseudo random pattern is projectedonto a corresponding position on the patient/target. The image sensormay capture a 2D intraoperative image of a plurality of object pointsassociated with the patient/target, to calculate the 3D topography.

The controller 310 may associate each object point associated thepatient that is captured by the image sensor with a corresponding dotincluded in the pseudo random pattern that is projected onto thepatient/target by the projector based on the position of eachcorresponding dot as pre-determined by the projector. The controller 310may convert the 2D image to the 3D scan of the patient/target based onthe association of each object point to each position of eachcorresponding dot included in the pseudo random pattern aspre-determined by the projector. In one example, the projector mayinclude one or more edge emitting laser, at least one collimating lens,and at least one diffractive optics element. The edge emitting laser andthe diffractive optics element may be controlled by the controller 310to generate patterns desirable for the specific 3D scanningapplications.

It should be appreciated that the near eye 3D display may comprise LCD(liquid crystal) microdisplays, LED (light emitting diode)microdisplays, organic LED (OLED) microdisplays, liquid crystal onsilicon (LCOS) microdisplays, retinal scanning displays, virtual retinaldisplays, optical see through displays, video see through displays,convertible video-optical see through displays, wearable projectiondisplays, and the like. In another example, the digital magnificationwearable device configuration 800 may further include a light source forsurgical field illumination. In one example, the light source is basedon one or a plurality of light emitting diode (LED). In another example,the light source is based on one or a plurality of laser diode withwaveguide or optical fiber. In another example, the light source has adiffuser. In another example, the light source has noncoherent lightsource such as an incandescent lamp. In yet another example, the lightsource has coherent light source such as a laser diode andphosphorescent materials in film form or volumetric form. In yet anotherembodiment, the light source is mounted on a surgical instrument toillumination of the cavity.

In another embodiment, the image sensors 330(a-b) are a pair ofmonochrome sensors. The systems further include a least one fluorescenceemission filter. Thus, the digital magnification surgical loupeconfiguration may digitally magnify stereoscopic fluorescence images anddisplay to the user in the near-eye 3D display 320(a-b) in 3D. Thesystems further include a light source that is capable of provideexcitation light to the surgical field. It should also be appreciatedthat the light source may include a laser light; a light emitting diode(LED); an incandescent light; a projector lamp; an arc-lamp, such asxenon, xenon mercury, or metal halide lamp; as well as coherent orin-coherent light sources. In one example, the light source comprises ofone or a plurality of white LEDs with a low pass filter (e.g. 775 nmshort pass filter) and one or a plurality of near infrared LEDs with aband pass filter (e.g. 830 nm band pass filter). In another example, thelight source comprises of one or a plurality of white LEDs with a lowpass filter (e.g. 775 nm short pass filter) and one or a plurality ofnear infrared LEDs with a long pass filter (e.g. 810 nm long passfilter). In one example, the light source can be controlled by sensorssuch as an inertial measurement unit to turn the light on and off.

In another embodiment, the digital magnification wearable deviceconfiguration 800 includes at least two color image sensors, at leasttwo monochrome image sensors, at least two beamsplitters, and at leasttwo narrow band filters. The monochrome image sensor, the color sensorand the beamsplitter are optically aligned on each side (left vs right),so that the left color image is aligned with the left monochrome image,and the right color image is aligned with the right monochrome image. Itshould be appreciated that the beamsplitters can be cube beamsplitters,plate beamsplitters, Pellicle Beamsplitters, Dichroic Beamsplitters, orpolarizing beamsplitters. It should be appreciated that the opticaldesign can be in a folded configuration using mirrors.

In another example, the digital magnification wearable deviceconfiguration 800 includes a light source with an additional spectralfilter. The digital magnification wearable device configuration 800 maybe used to capture narrow band reflectance images or fluorescenceimages, and to digitally magnify the image and display to the user in 3Dwith desirable binocular overlap. For example, the light source may be aplurality of white LEDs and near infrared LEDs (770 nm), and thespectral filter can be a 800 nm short pass filter. In anotherembodiment, the apparatus further includes additional sensors, such asan inertial measurement unit (IMU), accelerometers, gyroscopes,magnetometers, proximity sensors, microphone, force sensors, ambientlight sensors, etc. In one example, the light source can be controlledby sensors such as an inertial measurement unit to turn the light on andoff. In another example, the system 300 can be controlled by sensorssuch as an inertial measurement unit and/or proximity sensor to turn thesystem 300 on and off. Some example of types of proximity sensors are:Photoelectric, Inductive, Capacitive and Ultrasonic.

In one embodiment, the digital magnification wearable deviceconfiguration 800 further include at least one microphone. The system300 may record audio data such as dictation. The system 300 capture theaudio data using the microphone, perform voice recognition on thecontroller 310, and enable voice control of the system 300. In oneaspect, the voice control may include adjustment of the magnificationlevels (e.g. from 3× to 5×). In one example, a microphone array ormultiple microphones are used, the system may triangulate the source ofsound for multiple purposes such as noise cancellation, voice control ofmultiple devices in close proximity, etc. The system 300 maydifferentiate the one user from other users based on the triangulationof voice/audio signal. In yet another embodiment, the digitalmagnification wearable device configuration 800 further includestracking hardware, such as optical tracking hardware, electromagnetictracking hardware, etc. In yet another embodiment, the digitalmagnification wearable device configuration 800 further includes ofcommunication hardware, to enable wireless or wired communication suchas such as Wi-fi, Bluetooth, cellular communication, Ethernet, LAN,wireless communication protocols compatible with operating rooms,infrared communication. The apparatus can thus stream the magnificationdata and/or the original image data captured by the image sensors toanother apparatus, computer or mobile devices. In yet anotherembodiment, the lenses 340(a-b) in the digital magnification wearabledevice configuration 800 include autofocus lenses.

In yet another embodiment, the lenses 340(a-b) in the digitalmagnification wearable device configuration 800 are autofocus lenses butthe digital magnification wearable device configuration 800 may focusthe lenses, on request of the user. For example, upon user request viaan input device or via voice control, the lenses will be focused on thedemand of the user. Thus, the autofocus will not be activated unlessdemanded by the user, thus avoiding unwanted autofocus during surgicalprocedures. In one example, the focus setting of the left lens 340 b andright lens 340 a are always the same. For example, the settings forfocusing left lens 340 b and the settings for right lens 340 a are setto be the same, to avoid left lens focusing on a focal plane differentfrom the right plane.

In yet another embodiment, the digital magnification wearable deviceconfiguration 800 further includes additional input devices, such as afoot pedal, a wired or a wireless remote control, one or more button, atouch screen, microphone with voice control, gesture control device suchas Microsoft Kinect, etc. It should be appreciated that the controllercan be useable or disposable. It should be appreciated that a sterilesheet or wrap may be placed around the input device. In yet anotherembodiment, the digital magnification wearable device configuration 800may display medical images such as MRI (magnetic resonance image) imagedata, computed tomography (CT) image data, positron emission tomography(PET) image data, single-photon emission computed tomography (SPECT),PET/CT, SPECT/CT, PET/MRI, gamma scintigraphy, X-ray radiography,ultrasound, and the like. In yet another embodiment, the digitalmagnification wearable device configuration 800 may include digitalstorage hardware, to enable recording the magnification data, and/or theoriginal image data from image sensors, and/or audio data, and/or othersensor data.

Image Stabilization

In one example, electronic image stabilization (EIS) is implemented bythe Controller 310. The Controller 310 shifts the electronic image fromframe to frame of left video captured by the left camera and the rightvideo captured by the right camera, enough to counteract the motion. EISuses pixels outside the border of the cropped area during digitalmagnification to provide a buffer for the motion. In one aspect, opticalflow or other image processing methods may be used to track subsequentframes and detect vibrational movements and correct for them. In anotheraspect, feature-matching image stabilization methods may be used. Imagefeatures may be extracted via SIFT, SURF, ORB, BRISK, neural networks,etc.

In another example, Optical Image Stabilization (OIS) is implemented. Inone aspect, the OIS in the lenses 340 a and 340 b. For instance, usingsprings and mechanical mount, image sensor movements are smoothened orcancelled out. In one aspect, the image sensors 330 a and 330 b can bemoved in such a way as to counteract the motion of the camera.

In yet another example, mechanical image stabilization (MIS) isimplemented. Gimbals may be used for MIS. In one instance, MIS isachieved by attaching a gyroscope to the system. He gyroscope lets theexternal gyro (gimbal) stabilize the image sensors 330 a and 330 b.

Stereoscopic Calibration

The system 300 may need stereoscopic calibration to enable accurate 3Ddigital magnification. In one example, after mechanical fixture toachieve vertical calibration, a single calibration (through repeatedcapture of similar calibration pattern such as fiducials or chessboard)on left and right sensors, based on that an initial homographytransformation and cropping is applied to the pair of images to achievea high accuracy alignment between the two in executed. This is similarto finding the epipolar geometry between two sensors and bringing thetwo frames into a single plane through calibration to have: (1)Identical scales of the captured geometry, through virtual identicalfocal length, (2) Identical peripheral alignment of captured scene,through undistortion, and (3) Identical vertical alignment of capturedframes, through homography (projective) transformation. The newcalibrated frames (rectified frames) may be used for subsequent digital3D magnification and visualization processes, as previously described.

Ergonomic Calibration

In one aspect, ergonomic calibration can be performed on the system 300using one or a plurality of IMUs, one on the image sensor axis andsecond one on the display axis. Two important objectives are achieved incapturing and displaying the digital images: the headset is horizontallyaligned in the center of the forehead (single IMU reading andcorrection). This is essential to have a symmetrical mechanical positionfor each image sensor 330 a and 330 b with respect to each correspondingeye (left sensor 330 a to left eye and right sensor 330 b to right eye).It also helps achieve maintain binocular overlap between the digitallymagnified images captured and overlapped in the center of the two imagesensors (by comparing and aligning the two IMUs), and the center of thetwo eyes which is perceived by natural vision around the displays.

Autofocus and Autofocus On-Demand

Autofocus can be achieved through mechanical structure such asmotors/actuators or through liquid lenses. In one example, thecontroller 310 may conduct brightness assessment to find a high contrastimage, high frequency values, etc. through a method of Sobel filter orsimilar that extracts edges and high frequency features of the leftand/or right images. The autofocus lens may test a large range of focus(course focus) to find a course focus, and subsequently conduct asmaller range of focus (fine focus) based in the neighborhood near thecourse focus. In one example, the right lens 340 a and left lens 340 bmay be assigned to 2 ends of the focus range and progress towards themiddle. Once the an optical focus value is found, both lenses willassigned the same value or similar value, to avoid 2 lenses focusing ondifferent image planes.

In another example, the controller 310 may conduct using calibration anddisparity map to find the working distance of desired object. Thecontroller 310 may use previously calibrated frames to extract a partialor full disparity or depth map. Then controller 310 may use a region ofinterest or a point in a specific part of the image to assess thedistance to the desired object or plane of operation (working distance),and use the distance to determine proper value for autofocus from eithera distance dependent equation or a pre-determined look-up-table (LUT).

Additional Methods to Maintain Binocular Overlap During DigitalMagnification

The binocular overlap may be defined as a variable of working distanceand magnification level. By detecting and calculating the workingdistance of the patient/target from the image sensors 330 a and 330 b,the controller 310 can defining the proper value for binocular overlapbetween binocular views to achieve proper 3D visualization, from eithera distance-dependent equation or a pre-determined look-up-table (LUT)after defining the distance to the point of interest or average workingdistance of the region of interest. In one instance, distance can beinferred using calibration and disparity map to find distance. Usingpreviously calibrated frames to extract a partial or full disparity ordepth map (the two are related but different numerical values). Then thecontroller 310 may use a region of interest or a point in a specificpart of the image to extract the distance to the desired object or planeof operation (working distance), In another instance, the controller 310may use autofocus values of left autofocus lens and/or right autofocuslens to inter the working distance.

Controller

The controller 310 comprises the hardware and software necessary toimplement the aforementioned methods. In one embodiment, the controller310 involves a computer-readable medium comprising processor-executableinstructions configured to implement one or more of the techniquespresented herein. An example embodiment of a computer-readable medium ora computer-readable device comprises a computer-readable medium, such asa SSD, CD-R, DVD-R, flash drive, a platter of a hard disk drive, etc.,on which is encoded computer-readable data. This computer-readable data,such as binary data comprising at least one of a zero or a one, in turncomprises a set of computer instructions configured to operate accordingto one or more of the principles set forth herein. In some embodiments,the set of computer instructions are configured to perform a method,such as at least some of the exemplary methods described herein, forexample. In some embodiments, the set of computer instructions areconfigured to implement a system, such as at least some of the exemplarysystems described herein, for example. Many such computer-readable mediaare devised by those of ordinary skill in the art that are configured tooperate in accordance with the techniques presented herein.

The following discussion provide a brief, general description of asuitable computing environment to implement embodiments of one or moreof the provisions set forth herein. Example computing devices include,but are not limited to, personal computers that may comprise a graphicsprocessing unit (GPU), server computers, hand-held or laptop devices,mobile devices (such as mobile phones, Personal Digital Assistants(PDAs), media players, and the like), multiprocessor systems, consumerelectronics, mini computers, mainframe computers, a microcontroller, aField Programmable Gate Array (FPGA), an application-specific integratedcircuit (ASIC), distributed computing environments that include any ofthe above systems or devices, and the like. In one aspect, thecontroller may use a heterogeneous computing configuration.

Although not required, embodiments are described in the general contextof “computer readable instructions” being executed by one or morecomputing devices. Computer readable instructions may be distributed viacomputer readable media. Computer readable instructions may beimplemented as program components, such as functions, objects,Application Programming Interfaces (APIs), data structures, and thelike, that perform particular tasks or implement particular abstractdata types. Typically, the functionality of the computer readableinstructions may be combined or distributed as desired in variousenvironments.

In one example, a system comprises a computing device configured toimplement one or more embodiments provided herein. In one configuration,the computing device includes at least one processing unit and onememory unit. Depending on the exact configuration and type of computingdevice, the memory unit may be volatile (such as RAM, for example),non-volatile (such as ROM, flash memory, etc., for example) or somecombination of the two. In other embodiments, the computing device mayinclude additional features and/or functionality. For example, thecomputing device may also include additional storage (e.g., removableand/or non-removable) including, but not limited to, cloud storage,magnetic storage, optical storage, and the like. In one embodiment,computer readable instructions to implement one or more embodimentsprovided herein may be in the storage. The storage may also store othercomputer readable instructions to implement an operating system, anapplication program, and the like. Computer readable instructions may beloaded in the memory for execution by the processing unit, for example.

The term “computer readable media” as used herein includes computerstorage media. Computer storage media includes volatile and nonvolatile,removable and non-removable media implemented in any method ortechnology for storage of information such as computer readableinstructions or other data. Computer storage media includes, but is notlimited to, RAM, ROM, EEPROM, flash memory or other memory technology,CD-ROM, Digital Versatile Disks (DVDs) or other optical storage,magnetic disk storage or other magnetic storage devices, or any othermedium which can be used to store the desired information and which canbe accessed by computing device.

The computing device may also include communication connection(s) thatallows the computing device to communicate with other devices.Communication connection(s) may include, but is not limited to, a modem,a Network Interface Card (NIC), an integrated network interface, a radiofrequency transmitter/receiver, an infrared port, a USB connection, orother interfaces for connecting computing device to other computingdevices. Communication connection(s) may include a wired connection or awireless connection. Communication connection(s) may transmit and/orreceive communication media.

The computing device may include input device(s) such as keyboard,mouse, pen, voice input device, touch input device, infrared cameras,depth cameras, touchscreens, video input devices, and/or any other inputdevice. Output device(s) such as one or more displays, speakers,printers, and/or any other output device may also be included in thecomputing device. Input device(s) and output device(s) may be connectedto the computing device via a wired connection, wireless connection, orany combination thereof. In one embodiment, an input device or an outputdevice from another computing device may be used as input device(s) oroutput device(s) for computing device.

Components of computing device 6712 may be connected by variousinterconnects, such as a bus. Such interconnects may include aPeripheral Component Interconnect (PCI), such as PCI Express, aUniversal Serial Bus (USB), firewire (IEEE 1394), an optical busstructure, and the like. In another embodiment, components of computingdevice may be interconnected by a network. For example, the memory maybe comprised of multiple physical memory units located in differentphysical locations interconnected by a network.

Those skilled in the art will realize that storage devices utilized tostore computer readable instructions may be distributed across anetwork. For example, a computing device accessible via a network maystore computer readable instructions to implement one or moreembodiments provided herein. Computing device may access anothercomputing device and download a part or all of the computer readableinstructions for execution. Alternatively, the first computing devicemay download pieces of the computer readable instructions, as needed, orsome instructions may be executed at the first computing device and someat the second computing device.

Various operations of embodiments are provided herein. In oneembodiment, one or more of the operations described may constitutecomputer readable instructions stored on one or more computer readablemedia, which if executed by a computing device, will cause the computingdevice to perform the operations described. The order in which some orall of the operations are described should not be construed as to implythat these operations are necessarily order dependent. Alternativeordering will be appreciated by one skilled in the art having thebenefit of this description. Further, it will be understood that not alloperations are necessarily present in each embodiment provided herein.Also, it will be understood that not all operations are necessary insome embodiments.

Conclusion

It is to be appreciated that the Detailed Description section, and notthe Abstract section, is intended to be used to interpret the claims.The Abstract section may set forth one or more, but not all exemplaryembodiments, of the present disclosure, and thus, is not intended tolimit the present disclosure and the appended claims in any way.

The present disclosure has been described above with the aid offunctional building blocks illustrating the implementation of specifiedfunctions and relationships thereof. The boundaries of these functionalbuilding blocks have been arbitrarily defined herein for the convenienceof the description. Alternate boundaries may be defined so long as thespecified functions and relationships thereof are appropriatelyperformed.

It will be apparent to those skilled in the relevant art(s) the variouschanges in form and detail may be made without departing from the spirtand scope of the present disclosure. Thus the present disclosure shouldnot be limited by any of the above-described exemplary embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A system for generating three-dimensional (3D)images from captured images of a target when executing digitalmagnification on the captured images to maintain the 3D images generatedof the target after digital magnification, comprising: a first imagesensor that is configured to capture a first image of the target; asecond image sensor is configured to capture a second image of thetarget; a controller configured to: execute a digital magnification onthe first image captured by the first image sensor and on the secondimage captured by the second image sensor, crop the first image and thesecond image to overlap a first portion of the target captured by thefirst image sensor with a second portion of the target captured by thesecond image sensor, wherein the first portion of the target overlapswith the second portion of the target, adjust the cropping of the firstimage and the second image to provide binocular overlap of the firstportion of the target with the second portion of the target, wherein thebinocular overlap of the first image and the second image is an overlapthreshold that when satisfied results in a 3D image of the targetdisplayed to a user after the digital magnification is executed, andinstruct a display to display the cropped first image and the croppedsecond image that includes the binocular overlap to the user, whereinthe displayed cropped first image and the cropped second image displaythe 3D image at the digital magnification to the user.
 2. The system ofclaim 1, wherein the controller is further configured to: resize thecropped first image to the original size of the first image captured bythe first image sensor and the cropped second image to the original sizeof the second image captured by the second image sensor, wherein thecropped first image as resized and the cropped second image as resizedincludes the binocular overlap of the first image and the second image;and instruct the display to display the resized and cropped first imageand the resized and cropped second image that includes the binocularoverlap to the user, wherein the displayed resized and cropped firstimage and the resized and cropped second image display the 3D image atthe digital magnification to the user.
 3. The system of claim 2, whereinthe controller is further configured to: crop the first image capturedby the first image sensor and the second image captured by the secondimage sensor to vertically align the overlap of the first portion of thetarget with the second portion of the target, wherein the cropped firstimage is in vertical alignment of the cropped second image when a firstplurality of vertical coordinates of the cropped first image is alignedwith each corresponding vertical coordinate from a second plurality ofcoordinates of the cropped second image; adjust the cropping of thefirst image and the second image to provide binocular overlap of thefirst portion of the target with the second portion of the target,wherein the binocular overlap of the first image and the second image isvertically aligned to satisfy the overlap threshold to generate the 3Dimage of the target displayed to the user after the digitalmagnification is executed.
 4. The system of claim 3, wherein thecontroller is further configured to: after executing a first digitalmagnification at a first digital magnification level on the first imagecaptured by the first image sensor and on the second image captured bythe second image sensor, maintain the binocular overlap generated byadjusting the cropping of the first image and the second image tosatisfy the overlap threshold; execute a second digital magnification ata second digital magnification level on the first image captured by thefirst image sensor and on the second image captured by the second imagesensor, wherein the second digital magnification level is increased fromthe first digital magnification level; and maintain the binocularoverlap generated after executing the first digital magnification at thefirst digital magnification level on the first image and the secondimage to when executing the second digital magnification at the seconddigital magnification level.
 5. The system of claim 4, wherein thecontroller is further configured to: after executing each previousdigital magnification at each previous digital magnification level onthe first image and the second image, maintain the binocular overlap andthe vertical alignment determined when executing the first digitalmagnification at the first digital magnification level on the firstimage and the second image; and continue to maintain the binocularoverlap and the vertical alignment determined from the adjusting of thecropping of the first image and the second image to satisfy the overlapthreshold after executing the first digital magnification at the firstdigital magnification level on the first image and the second image foreach subsequent digital magnification at each subsequent digitalmagnification level, wherein each subsequent digital magnification levelis increased from each previous digital magnification level.
 6. Thesystem of claim 4, wherein the controller is further configured to:execute the first digital magnification at the first digitalmagnification level on a non-concentric portion of the first image and anon-concentric portion of the second image, wherein the non-concentricportion of the first image and the second image is a portion of thefirst image and the second image that differs from a center of the firstimage and the second image; adjust the cropping of the first image andthe second image to provide binocular overlap of the non-concentricportion of the first image and the non-concentric portion of the secondimage, wherein the binocular overlap of the non-concentric portion ofthe first image and the non-concentric portion of the second imagesatisfies the overlap threshold; and continue to crop a non-concentricportion of the first image and a non-concentric portion of the secondimage for each subsequent digital magnification at each subsequentdigital magnification level, wherein the binocular overlap of thenon-concentric portion of the first image and the non-concentric portionof the second image is maintained from the first digital magnificationat the first digital magnification level.
 7. The system of claim 4,wherein the controller is further configured to: determine a distancethat the first image sensor and the second image sensor is positionedfrom the target; execute the cropping of the first image and the secondimage to maintain the vertical alignment and the binocular overlap foreach digital magnification at each digital magnification level based onthe distance of the first image sensor and the second image sensor fromthe target.
 8. The system of claim 4, further comprising at least onewearable display that displays the resized and cropped first image andthe resized and cropped second image to display the 3D image of thetarget after the digital magnification is executed that includes thebinocular overlap of the first image and the second image that arevertically aligned to satisfy the overlap threshold.
 9. The system ofclaim 4, further comprising a display that is configured to: display theresized and cropped first image and the resized and cropped second imageto thereby display the 3D image of the target after the digitalmagnification is executed that includes the binocular overlap of thefirst image and the second image that are vertically aligned to satisfythe overlap threshold.
 10. The system of claim 5, wherein the overlapthreshold is satisfied when the binocular overlap includes 75% overlapof the first image and the second image and is maintained for eachsubsequent digital magnification at each subsequent digitalmagnification level.
 11. A method for generating a three-dimensional(3D) images from captured images of a target when executing digitalmagnification on the captured images to maintain the 3D images generatedof the target after digital magnification, comprising: capturing a firstimage by a first image sensor of the target; capturing a second image bya second image sensor of the target; executing by a controller a digitalmagnification on the first image captured by the first image sensor ofthe target and the second image captured by the second image sensor ofthe target; cropping the first image and the second image to overlap afirst portion of the target captured by the first image sensor with asecond portion of the target captured by the second image sensor,wherein the first portion of the target overlaps partially or fully withthe second portion of the target; adjusting the cropping of the firstimage and the second image to provide binocular overlap of the firstportion of the target with the second portion of the target, wherein thebinocular overlap of the first image and the second image is an overlapthreshold that when satisfied results in a 3D image of the targetdisplayed to a user after the digital magnification is executed; andinstructing a display to display the cropped first image and the croppedsecond image that includes the binocular overlap to the user, whereinthe displayed cropped first image and the cropped second image displaythe 3D image at the digital magnification to the user.
 12. The method ofclaim 11, further comprising: resizing the cropped first image to theoriginal size of the first image captured by the first image sensor andthe cropped second image to the original size of the second imagecaptured by the second image sensor, wherein the cropped first image asresized and the cropped second image as resized includes the binocularoverlap of the first image and the second image; and instructing thedisplay to display the resized and cropped first image and the resizedand cropped second image that includes the binocular overlap to theuser, wherein the displayed resized and cropped first image and theresized and cropped second image display the 3D image at the digitalmagnification to the user.
 13. The system of claim 12, furthercomprising: cropping the first image captured by the first image sensorand the second image captured by the second image sensor to verticallyalign the overlap of the first portion of the target with the secondportion of the target, wherein the cropped first image is in verticalalignment of the cropped second image when each vertical coordinate ofthe cropped first image is aligned with each corresponding verticalcoordinate of the cropped second image; and adjusting the cropping ofthe first image and the second image to provide binocular overlap of thefirst portion of the target with the second portion of the target,wherein the binocular overlap of the first image and the second image isvertically aligned to satisfy the overlap threshold to generate the 3Dimage of the target displayed to the user after the digitalmagnification is executed.
 14. The method of claim 13, furthercomprising: after executing a first digital magnification at a firstdigital magnification level on the first image captured by the firstimage sensor and on the second image captured by the second imagesensor, locking in the binocular overlap generated by adjusting thecropping of the first image and the second image to satisfy the overlapthreshold; executing a second digital magnification at a second digitalmagnification level on the first image captured by the first imagesensor and on the second image captured by the second image sensor,wherein the second digital magnification level is increased from thefirst digital magnification level; and maintaining the binocular overlapgenerated after executing the first digital magnification at the firstdigital magnification level on the first image and the second image whenexecuting the second digital magnification at the second digitalmagnification level.
 15. The method of claim 14, further comprising:after executing each previous digital magnification at each previousdigital magnification level on the first image and the second image,maintaining the binocular overlap and the vertical alignment determinedwhen executing the first digital magnification at the first digitalmagnification level on the first image and the second image; andcontinuing to maintain the binocular overlap and the vertical alignmentdetermined from the adjusting of the cropping of the first image and thesecond image to satisfy the overlap threshold after executing the firstdigital magnification at the first digital magnification level on thefirst image and the second image for each subsequent digitalmagnification at each subsequent digital magnification level, whereineach subsequent digital magnification level is increased from eachprevious digital magnification level.
 16. The method of claim 14,further comprising: executing the first digital magnification at thefirst digital magnification level on a non-concentric portion of thefirst image and on a non-concentric portion of the second image, whereinthe non-concentric portion of the first image and the second image is aportion of the first image and the second image that differs from acenter of the first image and the second image; adjusting the croppingof the first image and the second image to provide binocular overlap ofthe non-concentric portion of the first image and the non-concentricportion of the second image, wherein the binocular overlap of thenon-concentric portion of the first image and the non-concentric portionof the second image satisfies the overlap threshold; and continuing tocapture a non-concentric portion of the first image and a non-concentricportion of the second image for each subsequent digital magnification ateach subsequent digital magnification level, wherein the binocularoverlap of the non-concentric portion of the first image and thenon-concentric portion of the second image is maintained from the firstdigital magnification at the first digital magnification level.
 17. Themethod of claim 14, further comprising: determining a distance that thefirst image sensor and the second image sensor is positioned from thetarget; and executing the cropping of the first image and the secondimage to maintain the vertical alignment and the binocular overlap fordigital magnification at a digital magnification level based on thedistance of the first image sensor and the second image sensor from thetarget.
 18. The method of claim 14, further comprising: displaying by awearable display the resized and cropped first image and the resized andcropped second image to display the 3D image of the target after thedigital magnification is executed that includes the binocular overlap ofthe first image and the second image that are vertically aligned tosatisfy the overlap threshold.
 19. The method of claim 18, furthercomprising: positioning the first image sensor and the second imagesensor on the wearable display for the user to execute a surgicalprocedure on a target that is a patient.
 20. The method of claim 15,further comprising: satisfying the overlap threshold when the binocularoverlap includes overlap of the first image and the second image and ismaintained for each subsequent digital magnification at each subsequentdigital magnification level.